Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add uv publish: Basic upload with username/password or keyring #7475

Merged
merged 16 commits into from
Sep 24, 2024
Merged

Add uv publish: Basic upload with username/password or keyring #7475

merged 16 commits into from
Sep 24, 2024

Conversation

konstin
Copy link
Member

@konstin konstin commented Sep 17, 2024 •
edited
Loading

The uv publish command allows uploading wheels and source distributions to PyPI or another registry. We support username/password auth, token auth and (follow-up) trusted publishing from GitHub Actions. Be default we upload files from dist/*, the output directory from uv build.

This is intended to support three different workflow, ordered by relevance:

  1. Publishing a pure Python package from CI
  • Call uv build
  • Clear the venv, install the wheel, run a smoke test
  • Clear the venv, install the source distribution, run a smoke test
  • Call uv publish
  1. Publishing a native module package from CI
  • In one job, call uv build, clear the venv, install the source distribution, run a smoke test, upload the source distribution as artifact
  • For each target, run a job, call uv build --wheel, clear the venv, install the wheel run a smoke test, upload the wheel as artifact
  • In a final job, download all artifacts to dist/* and run uv publish
  1. Publishing from a developer machine
  • Call uv build
  • Do a manual test routine
  • Call uv publish

Since we intend for smoke tests between build and publish, there is no combined build-and-publish command.

What works:

  • Basic upload to PyPI with token or username/password (username must be __token__, but it uses the same HTTP headers)
  • Uploads to gitlab.com (tested with a personal access token)
  • Keyring integration. If you use scoped tokens, you need use url-string hacks such as Not obvious how to use multiple project API tokens with keyring pypa/twine#565 (comment) to differentiate the tokens, a problem we share with twine.
  • Error messages
  • Testing on CI. Every time something in the upload crate or its test script changes, we upload a new version of dummy packages to testpypi

This PR has coverage for PyPI (canonical index) and gitlab.com (alternative index). Unless there are concerns for a specific other index, I wouldn't add any specific testing for them yet.

Next PRs (all stacked on this one directly):

Follow-ups:

  • A demo repo with pure Python package that demonstrates scenario 1 in a copy&paste-able GitHub actions configuration. We can only demo this live once the feature shipped.

Not Implemented:

  • Prompting for passwords.

Best reviewed commit-by-commit

@konstin konstin added the enhancement New feature or improvement to existing functionality label Sep 17, 2024
konstin
konstin commented Sep 17, 2024
View reviewed changes
Cargo.lock Outdated Show resolved Hide resolved
@zanieb
Copy link
Member

zanieb commented Sep 17, 2024

I have a test script for test PyPI for both username/password and keyring (next PR), but we need to run it in CI in a secure way.

Does it need to be secure? Or can we just have a test user that only uploads dummy packages that we can expose the password to?

@konstin konstin force-pushed the konsti/publish2 branch 3 times, most recently from ea8fc61 to 3775e2c Compare September 18, 2024 21:26
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
49aa8f6 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
e9571d5 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
8a54b24 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
0debcb3 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
@konstin konstin mentioned this pull request Sep 19, 2024
Merged
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
fbc0482 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
@konstin konstin force-pushed the konsti/publish2 branch 3 times, most recently from 7502383 to 6f83d3f Compare September 19, 2024 12:45
konstin added a commit that referenced this pull request Sep 19, 2024
@konstin
Add only_authenticated option to the client
593fca0 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
lucab
lucab reviewed Sep 20, 2024
View reviewed changes
crates/uv-publish/src/lib.rs Outdated Show resolved Hide resolved
crates/uv/src/commands/publish.rs Outdated Show resolved Hide resolved
crates/uv-publish/src/lib.rs Outdated Show resolved Hide resolved
crates/uv/src/commands/publish.rs
.only_authenticated(true)
.build();

for (file, filename) in files {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to try making parallel uploads here, or do you think that is not worth it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently not planning to, but i made the upload async to make it easier to do in parallel (with other uploads or something else entirely) in the future.

konstin added a commit that referenced this pull request Sep 21, 2024
@konstin
Add only_authenticated option to the client
8f06d87 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
konstin added a commit that referenced this pull request Sep 21, 2024
@konstin
Add only_authenticated option to the client
1080964 
When sending an upload request, we use HTTP formdata requests, which can't be cloned (seanmonstar/reqwest#2416, plus a limitation that formdata bodies are always internally streaming), but also know that we need to always have credentials.

The authentication middleware by default tries to clone the request and send an authenticated request first. By introducing an `only_authenticated` setting, we can skip this behaviour for publishing.

Split out from #7475
@konstin konstin changed the base branch from main to konsti/only_authenticated September 21, 2024 12:44
@konstin konstin changed the base branch from main to konsti/split-metadata-parsing September 24, 2024 14:46
Base automatically changed from konsti/split-metadata-parsing to main September 24, 2024 15:16
konstin and others added 16 commits September 24, 2024 17:23
@konstin
Split metadata parsing
bb7bbd3 
In preparation to vendoring Metadata23 from https://github.com/PyO3/python-pkginfo-rs/blob/main/src/metadata.rs for #7475 (review)
@konstin
Add uv publish cli and configuration
9b784f3
@konstin
Implement uv publish
580ae7b
@konstin
CI integration testing for uv publish
81712e0
@konstin
uv publish is in preview
e23d31a
@konstin @charliermarsh
Update crates/uv-cli/src/lib.rs
abcea52 
Co-authored-by: Charlie Marsh <[email protected]>
@konstin @charliermarsh
Update crates/uv-cli/src/lib.rs
0f0d5b6 
Co-authored-by: Charlie Marsh <[email protected]>
@konstin @charliermarsh
Update crates/uv-cli/src/lib.rs
525be8d 
Co-authored-by: Charlie Marsh <[email protected]>
@konstin
Review message
a012a04
@konstin
Typo
da1b09c
@konstin
Review
060e002
@konstin
Snapshots
69cd646
@konstin
Use vendored METADATA parsing
2ec1367
@konstin
generate-all
d8ec252
@konstin
Use testpypi for tests
42684ae
@konstin
Use testpypi for tests and other review
7eed9bd
@konstin konstin enabled auto-merge (squash) September 24, 2024 15:28
@konstin konstin merged commit 1995d20 into main Sep 24, 2024
60 checks passed
@konstin konstin deleted the konsti/publish2 branch September 24, 2024 15:33
@BrewTestBot BrewTestBot mentioned this pull request Sep 24, 2024
Merged
@davidlrobinson davidlrobinson mentioned this pull request Sep 26, 2024
Open
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Oct 7, 2024
@tmeijn
build(deps): update astral-sh/uv to v0.4.18
f9ced3a 
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.15` -> `0.4.18` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.18`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0418)

[Compare Source](astral-sh/uv@0.4.17...0.4.18)

##### Enhancements

-   Allow multiple source entries for each package in `tool.uv.sources` ([#&#8203;7745](astral-sh/uv#7745))
-   Add `.gitignore` file to `uv build` output directory ([#&#8203;7835](astral-sh/uv#7835))
-   Disable jemalloc on FreeBSD ([#&#8203;7780](astral-sh/uv#7780))
-   Respect `PAGER` env var when paging in `uv help` command ([#&#8203;5511](astral-sh/uv#5511))
-   Support `uv run -m foo` to run a module ([#&#8203;7754](astral-sh/uv#7754))
-   Use a top-level output directory for `uv build` in workspaces ([#&#8203;7813](astral-sh/uv#7813))
-   Update `uv init --package` command to match project name ([#&#8203;7670](astral-sh/uv#7670))
-   Add a custom suggestion for `uv add dotenv` ([#&#8203;7799](astral-sh/uv#7799))
-   Add detailed errors for `tool.uv.sources` deserialization failures ([#&#8203;7823](astral-sh/uv#7823))
-   Improve error message copy for failed builds ([#&#8203;7849](astral-sh/uv#7849))
-   Use `serde-untagged` to improve some untagged enum error messages ([#&#8203;7822](astral-sh/uv#7822))
-   Use build failure hints for `dotenv` errors, rather than in `uv add` ([#&#8203;7825](astral-sh/uv#7825))

##### Configuration

-   Add `UV_NO_SYNC` environment variable ([#&#8203;7752](astral-sh/uv#7752))

##### Bug fixes

-   Accept `git+` prefix in `tool.uv.sources` ([#&#8203;7847](astral-sh/uv#7847))
-   Allow spaces in path requirements ([#&#8203;7767](astral-sh/uv#7767))
-   Avoid reusing cached downloaded binaries with `--no-binary` ([#&#8203;7772](astral-sh/uv#7772))
-   Correctly trims values during wheel WHEEL file parsing ([#&#8203;7770](astral-sh/uv#7770))
-   Fix `uv tree --invert` for platform dependencies ([#&#8203;7808](astral-sh/uv#7808))
-   Fix encoding mismatch between python child process and uv ([#&#8203;7757](astral-sh/uv#7757))
-   Reject self-dependencies in `uv add` ([#&#8203;7766](astral-sh/uv#7766))
-   Respect `tool.uv.environments` for legacy virtual workspace roots ([#&#8203;7824](astral-sh/uv#7824))
-   Retain empty extras on workspace members ([#&#8203;7762](astral-sh/uv#7762))
-   Use file stem when parsing cached wheel names ([#&#8203;7773](astral-sh/uv#7773))

##### Rust API

-   Make `FlatDistributions` public ([#&#8203;7833](astral-sh/uv#7833))

##### Documentation

-   Fix table of contents sizing ([#&#8203;7751](astral-sh/uv#7751))
-   GitLab Integration documentation ([#&#8203;6857](astral-sh/uv#6857))
-   Update documentation to setup-uv@v3 ([#&#8203;7807](astral-sh/uv#7807))
-   Use `uv publish` instead of twine in docs ([#&#8203;7837](astral-sh/uv#7837))
-   Fix typo in `projects.md` ([#&#8203;7784](astral-sh/uv#7784))

### [`v0.4.17`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0417)

[Compare Source](astral-sh/uv@0.4.16...0.4.17)

##### Enhancements

-   Add `uv build --all` to build all packages in a workspace ([#&#8203;7724](astral-sh/uv#7724))
-   Add support for `uv init --script` ([#&#8203;7565](astral-sh/uv#7565))
-   Add support for upgrading build environment for installed tools (`uv tool upgrade --python`) ([#&#8203;7605](astral-sh/uv#7605))
-   Initialize a Git repository in `uv init` ([#&#8203;5476](astral-sh/uv#5476))
-   Respect `--quiet` flag in `uv build` ([#&#8203;7674](astral-sh/uv#7674))
-   Add context message before listing available tools in `uvx` ([#&#8203;7641](astral-sh/uv#7641))

##### Bug fixes

-   Don't create Python bytecode files during interpreter discovery ([#&#8203;7707](astral-sh/uv#7707))
-   Escape glob patterns in workspace member discovery ([#&#8203;7709](astral-sh/uv#7709))
-   Avoid prefetching source distributions with unbounded lower-bound ranges ([#&#8203;7683](astral-sh/uv#7683))

##### Documentation

-   Add `uv build` and `uv publish` to features overview ([#&#8203;7716](astral-sh/uv#7716))
-   Add documentation on cache versioning ([#&#8203;7693](astral-sh/uv#7693))
-   Spell out the names of the Docker images for easier copy-paste ([#&#8203;7706](astral-sh/uv#7706))
-   Document uv-with-Jupyter workflows ([#&#8203;7625](astral-sh/uv#7625))
-   Note that `uv lock --upgrade-package` retains locked versions ([#&#8203;7694](astral-sh/uv#7694))

### [`v0.4.16`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0416)

[Compare Source](astral-sh/uv@0.4.15...0.4.16)

##### Enhancements

-   Add `uv publish` ([#&#8203;7475](astral-sh/uv#7475))
-   Add a `--project` argument to run a command from a project directory ([#&#8203;7603](astral-sh/uv#7603))
-   Display Python implementation when creating environments ([#&#8203;7652](astral-sh/uv#7652))
-   Implement trusted publishing for `uv publish` ([#&#8203;7548](astral-sh/uv#7548))
-   Respect lockfile preferences for `--with` requirements ([#&#8203;7627](astral-sh/uv#7627))
-   Unhide the `--directory` option ([#&#8203;7653](astral-sh/uv#7653))
-   Allow requesting free-threaded Python interpreters ([#&#8203;7431](astral-sh/uv#7431))
-   Show a dedicated PubGrub hint for `--unsafe-best-match` ([#&#8203;7645](astral-sh/uv#7645))
-   Add resolver error checking for conflicting distributions ([#&#8203;7595](astral-sh/uv#7595))

##### Bug fixes

-   Avoid adding double-newlines for CRLF ([#&#8203;7640](astral-sh/uv#7640))
-   Avoid retaining forks when `requires-python` range changes ([#&#8203;7624](astral-sh/uv#7624))
-   Determine if pre-release Python downloads should be allowed using the version specifiers ([#&#8203;7638](astral-sh/uv#7638))
-   Fix `link-mode=clone` for directories on Linux ([#&#8203;7620](astral-sh/uv#7620))
-   Improve Python executable name discovery when using alternative implementations ([#&#8203;7649](astral-sh/uv#7649))
-   Require opt-in to use alternative Python implementations ([#&#8203;7650](astral-sh/uv#7650))
-   Use the first pre-release discovered when only pre-release Python versions are available ([#&#8203;7666](astral-sh/uv#7666))

##### Documentation

-   Document environment variable that disables printing of virtual environment name in prompt ([#&#8203;7648](astral-sh/uv#7648))
-   Remove double whitespaces from the code ([#&#8203;7623](astral-sh/uv#7623))
-   Use anchorlinks rather than permalinks ([#&#8203;7626](astral-sh/uv#7626))

##### Preview features

-   Add build backend scaffolding ([#&#8203;7662](astral-sh/uv#7662))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zanieb zanieb zanieb left review comments

@charliermarsh charliermarsh charliermarsh approved these changes

@lucab lucab lucab left review comments

Assignees
No one assigned
Labels
enhancement New feature or improvement to existing functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@konstin @zanieb @lucab @charliermarsh